Early CRC delivery for partial frame

ABSTRACT

Memory apparatus and methods transmit and receive a CRC code for a first portion of a frame before the second portion of the frame is finished being transferred. The CRC may be used to check the first portion of the frame before the second portion of the frame is completely received. A command or other information in the first portion of the frame may be used without waiting for the rest of the frame. Other embodiments are described and claimed.

BACKGROUND

FIG. 1 illustrates a prior art memory system known informally asRamLink, which was proposed as a standard by the Institute of Electricaland Electronics Engineers (IEEE). The standard was designated as IEEEStd 1596.4-1996 and is known formally as IEEE Standard forHigh-Bandwidth Memory Interface Based on Scalable Coherent Interface(SCI) Signaling Technology (RamLink). The system of FIG. 1 includes amemory controller 10 and one or more memory modules 12. The memorycontroller 10 is typically either built into a processor or fabricatedon a companion chipset for a processor. Each memory module 12 has aslave interface 14 that has one link input and one link output. Thecomponents are arranged in a RamLink signaling topology known asRingLink with unidirectional links 16 between components. A controlinterface 18 on each module interfaces the slave interface 14 withmemory devices 20. In the system shown in FIG. 1, another RamLinksignaling topology known as SyncLink is used between the slaveinterfaces and memory devices.

The purpose of the RamLink system is to provide a processor withhigh-speed access to the memory devices. Data is transferred between thememory controller and modules in packets that circulate along theRingLink. The controller is responsible for generating all requestpackets and scheduling the return of slave response packets.

A write transaction is initiated when the controller sends a requestpacket including command, address, time, and data to a particularmodule. The packet is passed from module to module until it reaches theintended slave, which then passes the data to one of the memory devicesfor storage. The slave then sends a response packet, which is passedfrom module to module until it reaches the controller to confirm thatthe write transaction was completed.

A read transaction is initiated when the controller sends a requestpacket including command, address, and time to a module. The slave onthat module retrieves the requested data from one of the memory devicesand returns it to the controller in a response packet, which is againpassed from module to module until it reaches the controller.

FIG. 2 illustrates a prior art RamLink slave interface circuit. In thecircuit of FIG. 2, source-synchronous strobing is use to clock theincoming data signals. That is, a strobe signal, which accompanies theincoming data signals, is used to sample the incoming data. The circuitof FIG. 2 uses a phase-locked loop (PLL) to generate a stable localclock signal from a reference clock signal that is distributed to otherslave interface circuits. The local clock signal is used to reclock theoutgoing data signal so as to avoid cumulative jitter as the data ispassed along downstream.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a prior art RamLink memory system.

FIG. 2 illustrates a prior art RamLink slave interface circuit.

FIG. 3 illustrates an embodiment of a memory interface system accordingto the inventive principles of this patent.

FIG. 4 illustrates an embodiment of a memory module according to theinventive principles of this patent.

FIG. 5 illustrates another embodiment of a memory module and anembodiment of a memory buffer according to the inventive principles ofthis patent.

FIG. 6 illustrates additional example embodiments of a memory system,memory modules, and memory buffers according to the inventive principlesof this patent.

FIG. 7 illustrates another example embodiment of a memory bufferaccording to the inventive principles of this patent.

FIG. 8 illustrates an embodiment of a redrive circuit according to theinventive principles of this patent.

FIG. 9 illustrates an embodiment of an I/O cell according to theinventive principles of this patent.

FIG. 10 illustrates another embodiment of an I/O cell according to theinventive principles of this patent.

FIG. 11 illustrates another embodiment of an I/O cell according to theinventive principles of this patent.

FIG. 12 illustrates another embodiment of an I/O cell according to theinventive principles of this patent.

FIG. 13 illustrates an embodiment of a fail-over circuit according tothe inventive principles of this patent.

FIG. 14 illustrates another embodiment of a fail-over circuit operatingin a normal mode according to the inventive principles of this patent.

FIG. 15 illustrates another embodiment of a fail-over circuit operatingin a fail-over mode according to the inventive principles of thispatent.

FIG. 16 illustrates an embodiment of a memory buffer with bit lanefail-over capability according to the inventive principles of thispatent.

FIG. 17 illustrates an embodiment of a memory controller with bit lanefail-over capability according to the inventive principles of thispatent.

FIG. 18 illustrates an embodiment of a method for implementing permutingstatus patterns according to the inventive principles of this patent.

FIG. 19 illustrates an embodiment of a permuting pattern generator inaccordance with the inventive principles of this patent.

FIGS. 20 through 23 illustrate example embodiments of status patterns inaccordance with the inventive principles of this patent.

FIG. 24 illustrates an embodiment of a memory agent according to theinventive principles of this patent.

FIG. 25 illustrates an embodiment of a polling operation according tothe inventive principles of this patent.

FIG. 26 illustrates an embodiment of a memory module utilizing dataaccumulation according to the inventive principles of this patent.

FIG. 27 illustrates another embodiment of a memory module and anembodiment of a memory buffer utilizing data accumulation according tothe inventive principles of this patent.

FIG. 28 illustrates another example embodiment of a memory bufferutilizing data accumulation according to the inventive principles ofthis patent.

FIG. 29 illustrates an example embodiment of a frame according to theinventive principles of this patent.

FIG. 30 illustrates an example embodiment of a scheme for delivering aCRC code across multiple frames according to the inventive principles ofthis patent.

FIG. 31 illustrates an embodiment of a frame transfer schemeincorporating early CRC delivery and distributed CRC codes according tovarious inventive principles of this patent.

FIG. 32 illustrates an embodiment of a memory agent, in this exampleassumed to be a memory module or buffer, according to the inventiveprinciples of this patent.

FIG. 33 illustrates example embodiments of two possible bit lanemappings according to the inventive principles of this patent.

DETAILED DESCRIPTION

This patent encompasses numerous inventions that have independentutility. In some cases, additional benefits are realized when some ofthe principles are utilized in various combinations with one another,thus giving rise to additional inventions. These principles may berealized in myriad embodiments. Although some specific details are shownfor purposes of illustrating the inventive principles, numerous otherarrangements may be devised in accordance with the inventive principlesof this patent. Thus, the inventive principles are not limited to thespecific details disclosed herein.

FIG. 3 illustrates an embodiment of a memory interface system accordingto the inventive principles of this patent. The system of FIG. 3includes a memory controller 50 and one or more memory modules 52 thatcommunicate through a channel made up of unidirectional links. Thechannel has an outbound path that includes one or more outbound links54, and an inbound path that includes one or more inbound links 56. Eachmodule may be capable of redriving signals from link to link on theoutbound path and from link to link on the inbound path. Each module mayalso be capable of selectively disabling any redrive features, forexample, if the module detects that it is the outermost module, orresponsive to a command from the memory controller.

Each module includes one or more memory devices 58 arranged to transferdata to and/or from one or more of the paths. For example, the modulemay be arranged such that data from the outbound path is transferred toa memory device, and data from the memory device is transferred to theinbound path. One or more buffers may be disposed between one or morememory devices and one or more of the paths. The modules and controllerare not limited to any particular mechanical arrangement. For example,the modules may be fabricated on substrates separate from the rest ofthe system, they may be fabricated on a common substrate with thecontroller and links, or they may be realized in any other mechanicalarrangement. The modules are also not limited to any particular type ofmemory device, e.g., read only memory (ROM), dynamic random accessmemory (DRAM), flash memory, etc.

FIG. 4 illustrates an embodiment of a memory module according to theinventive principles of this patent. The module of FIG. 4 includes tworedrive circuits 60 and 62 to receive signals on unidirectional links54A and 56A, and redrive the signals on unidirectional links 54B and56B, respectively. One or more memory devices 58 are arranged totransfer data to and/or from one or more of the redrive circuits.

The module of FIG. 4 is not limited to any particular arrangement ofunidirectional links or any particular arrangement for transferring datato and/or from the redrive circuits. If the module of FIG. 4 is to beused in a memory system such as that shown in FIG. 3, then redrivecircuit 60 might be designated as an outbound redrive circuit andarranged to receive and redrive signals on an outbound path includinglinks 54A and 54B, and the other redrive circuit 62 might be designatedas an inbound redrive circuit and arranged to receive and redrivesignals on an inbound path including links 56A and 56B. In this example,one or more memory devices 58 may be arranged so that data istransferred from the outbound redrive circuit 60 to the memory device(s)and from the memory device(s) to the inbound redrive circuit 62.

The module may be capable of detecting if it is the outermost module ona channel and selectively disabling any redrive features accordingly.For example, if the module of FIG. 4 is to be used in a memory systemsuch as that shown in FIG. 3, and the module detects that it is theoutermost module, the outbound redrive circuit receives incoming signalson link 54A but does not redrive them. Likewise, the inbound redrivecircuit only drives link 56B with signals corresponding to data receivedfrom the memory device(s), and/or other signals that may be generatedinternally by the module. Alternatively, even if the module detects thatit is not the outermost module, it may be constructed so that it iscapable of operating as though it is the outermost module (for example,in response to a command from a memory controller), in which case it mayignore signals received on inbound link 56A, and it may not redrivesignals onto outbound link 54B.

FIG. 5 illustrates another embodiment of a memory module and anembodiment of a memory buffer according to the inventive principles ofthis patent. The module of FIG. 5 includes a memory buffer 64 having tworedrive circuits 60 and 62 to receive signals on unidirectional links54A and 56A, and redrive the signals on unidirectional links 54B and56B, respectively. The memory buffer also includes a memory interface 66arranged to transfer data to and from one or more memory devices 58. Thebuffer may be capable of detecting if it is the last agent on a channeland selectively disabling any redrive features accordingly. The buffermay be capable of operating as though it is the last agent on a channel,even if it is not, for example, responsive to a command from a memorycontroller. As used herein, agent refers to any memory controller (alsocalled a host), module, buffer, etc. that is interfaced to the channel.

The module and buffer of FIG. 5 are not limited to any particulararrangement of unidirectional links or any particular arrangement fortransferring between the memory interface and the redrive circuits. Ifthe module of FIG. 5 is to be used in a memory system such as that shownin FIG. 3, then redrive circuit 60 might be designated as an outboundredrive circuit and arranged to receive and redrive signals on anoutbound path including links 54A and 54B, and the other redrive circuit62 might be designated as an inbound redrive circuit and arranged toreceive and redrive signals on an inbound path including links 56A and56B. In this example, the memory interface may be configured to receivedata from the outbound redrive circuit 60 and to send data to theinbound redrive circuit 62.

Various mechanical arrangements may be used to implement the memorymodules and/or buffer of FIGS. 4 and 5. For example, the memory devices58, redrive circuits 60 and 62, and buffer 64 may all be realized asseparate integrated circuits mounted on a common circuit board or onseparate circuit boards. Various combinations of the components may befabricated together on a common integrated circuit, or they all might befabricated on a single integrated circuit. The circuit board or boards,if any, may be capable of being plugged into sockets on a motherboard,fabricated integral with a motherboard, or arranged in any other way.There may not be a circuit board, for example, if the components arefabricated as part of a multi-chip module. A memory buffer according tothe inventive principles of this patent may be used to interface devicesother than memory devices to a channel. For example a memory bufferaccording to the inventive principles of this patent may be used tointerface an I/O controller or a bridge to a channel.

Additional embodiments of apparatus according to the inventiveprinciples of this patent are described with reference to “inbound” and“outbound” paths, links, redrive circuits, etc. to facilitate anunderstanding of how the apparatus may be utilized in a memory systemsuch as the embodiment shown in FIG. 3. These apparatus, however, arenot limited to any particular arrangement of unidirectional links, tothe particular arrangements shown for transferring data between thelinks and other circuitry, or to any of the implementation detailsshown.

FIG. 6 illustrates additional example embodiments of a memory system,memory modules, and memory buffers according to the inventive principlesof this patent. Referring to FIG. 6, one or more memory modules 52 arebased on printed circuit boards having contact fingers along both sidesof one edge to create a dual inline memory module (DIMM) that may beplugged into a connector on another circuit board that holds othercomponents of the system. An existing form-factor may be adopted for themodule, for example the DIMM form-factor used for Double Data Rate II(DDR2) dynamic random access memory (DRAM) modules.

The modules are populated with memory devices 58, for example,commodity-type DRAM such as DDR2 DRAM. A memory buffer 64 on each moduleisolates the memory devices from a channel that interfaces the modulesto the memory controller 50, which is also referred to as a host. Thechannel is wired in a point-to-point arrangement with an outbound paththat includes outbound links 54, and an inbound path that includesinbound links 56. The links may be implemented with parallelunidirectional bit lanes using low-voltage differential signals.

In the embodiments of FIG. 6, no additional signal lines are used forfunctions such as command, reset, initialization, and the like. Instead,these functions are encoded directly in the data sent over the channel.Alternatively, however, any number of additional signal lines may beused to implement such functions.

A reference clock signal REF CLK is generated by a clock synthesizer 76distributed to the host and modules, maybe through a clock buffer 78.This facilitates a quasi-asynchronous clocking scheme in which locallygenerated clock signals are used to sample and redrive incoming data.Because a common reference clock is available at each agent, datasignals may be clocked without any frequency tracking. Alternatively, alocal clock signal may be generated independently of any referenceclock. As another alternative, a synchronous clocking scheme such assource synchronous strobing may be used.

In one possible embodiment, the host initiates data transfers by sendingdata, maybe in packets or frames (terms used interchangeably here), tothe innermost module on the outbound path. The innermost module receivesand redrives the data to the next module on the outbound path. Eachmodule receives and redrives the outbound data until it reaches theoutermost module. Although the outermost module could attempt to redrivethe data to a “nonexistent” outbound link, each module may be capable ofdetecting (or being instructed) that it is the outermost module anddisabling any redrive circuitry to reduce unnecessary power consumption,noise, etc. In this embodiment, data transfers in the direction of thehost, i.e., inbound, are initiated by the outermost module. Each modulereceives and redrives inbound data along the inbound path until itreaches the host.

Any suitable communication protocol may be used over the physicalchannel. For example, the host may be designated to initiate andschedule all inbound and outbound data transfers. Alternatively, anyagent may be allowed to initiate data transfers. Frames of data may beconfigured to carry commands, read data, write data, status information,error information, initialization data, idle patterns, etc., or anycombination thereof. A protocol may be implemented such that, when thehost sends a command frame to a target module along the outbound path,the target module responds by immediately sending a response frame backto the host along the inbound path. In such an embodiment, the targetmodule does not redrive the command frame on the outbound path.

In an alternative embodiment, the target module receives the commandframe and then redrives the command frame on the outbound path. When theoutermost module receives the command frame, it initiates a responseframe (maybe nothing more than an idle frame) on the inbound path. Thetarget module waits until the response frame reaches its inboundreceiver. The target module then mergers its response into the inbounddata stream, e.g., by replacing the response frame sent by the outermostmodule with the target module's true response frame.

FIG. 7 illustrates another example embodiment of a memory bufferaccording to the inventive principles of this patent. The memory bufferof FIG. 7 includes an outbound redrive circuit 60 to receive and redrivesignals on an outbound path including links 54A and 54B, and an inboundredrive circuit 62 to receive and redrive signals on an inbound pathincluding links 56A and 56B. A memory interface 66 interfaces the bufferto one or more memory devices, which may be through a memory bus 68. Thememory interface may include read and/or write buffers such as FIFObuffers. Data from the outbound path is coupled to the memory interface,which may be through a deskew circuit 70 which eliminates skew betweenbits of data if the outbound path has more than one bit lane. A patterngenerator 72 may be used to generate status patterns to transmit ontothe inbound path, for example, if the buffer happens to be the outermostagent on a channel, in which case, there may be no signals beingreceived on incoming inbound link 56A. A multiplexer 74 selectivelycouples data from the memory interface or pattern generator to theinbound redrive circuit.

The memory interface is not limited to any particular arrangement, andit may be compatible with standard memory devices, particularlycommodity memory devices such as DDR2 DRAM. The entire memory buffer maybe integrated on a single integrated circuit, it may be integrated intoone or more memory devices, its constituent elements may be integratedonto separate components, or any other mechanical arrangement may beemployed. The embodiment shown in FIG. 7 is exemplary only, and otherembodiments are possible in accordance with the inventive principles ofthis patent. For example, the embodiment of FIG. 7 is shown withunidirectional data flowing from the outbound redrive circuit to thememory interface and from the memory interface to the inbound redrivecircuit. This data flow, however, may be bi-directional, and otherarrangements are contemplated. Even if the embodiment of FIG. 7 is to beused in a channel system in which data for the memory interface onlyneeds to flow as shown in FIG. 7, it may still be realized with redrivecircuits having full bi-directional data access as this may facilitate,for example, implementation of built-in self-test (BIST) functions, inwhich case a second deskew circuit for deskewing data from the inboundpath may be helpful.

FIG. 8 illustrates an embodiment of a redrive circuit according to theinventive principles of this patent. The circuit of FIG. 8 includes oneor more input/output (I/O) cells 74, each of which receives an inputdata signal RX that it may redrive as an output data signal TX.Alternatively, an I/O cell may substitute or merge a read data signalRDX into the output data signal. A write data signal WDX may be takenfrom the input data signal, either before or after it is redriven as theoutput data signal.

The “X” in any of the above signal names indicates that it might be oneof multiple similar signals depending on the number of I/O cells in theredrive circuit. For example, a redrive circuit having nine bit laneswould have nine I/O cells with input data signals named R0, R1 . . . R8.In a redrive circuit with only a single I/O cell, the data input signalwould be R0 or simply R. The term RX is used to refer generically to anyor all of the input data signals.

The term “write data” is used for convenience to indicate any data beingtaken from the data stream traveling through the I/O cell. This does notimply, however, that write data must be directed to a memory interfaceor memory device. Likewise, “read data” refers to any data that is inputto the I/O cell, but read data may come from any source, not just amemory device or memory interface.

Referring again to FIG. 8, a clock generator 80 generates a number ofphase clock signals PCX and a transmit clock signal TC in response to areference clock signal REF CLK. The clock generator includes a phaselocked loop (PLL) 82 that generates the transmit clock TC as a multipleof the reference clock signal REF CLK, and a phase clock generator 84.In one possible embodiment, there are four phase clock signals PC0, PC1,PC2 and PC3 spaced 90 degrees apart and derived from the transmit clockTC. Each of the I/O cells may use one or more of the TC and PCX clocksignals to sample and/or redrive data signals, and/or to generateadditional local clock signals. In this embodiment, the phase clock andtransmit clock signals are stable signals in the sense that they are notadjusted in response to the phase of any of the input data signals RX.

FIG. 9 illustrates an embodiment of an I/O cell according to theinventive principles of this patent. A receiver 86 is arranged toreceive a data signal RX and redrive it as data signal TX in response toa sampling clock signal SC. The sampling clock signal is generated by asampling clock generator 88, which is capable of adjusting the samplingclock signal in response to the data signal RX. A write data signal WDXmay be taken from the input or the output of receiver 86. If taken fromthe output of the receiver as shown in FIG. 9, the sampling clock signalSC may be used as, or to derive, a strobe signal for the write data. Theinput to the sampling clock generator may be taken from points otherthan the input of the receiver as shown in FIG. 9. For example, it maybe taken from the output of the receiver as well.

FIG. 10 illustrates another embodiment of an I/O cell according to theinventive principles of this patent. In the embodiment of FIG. 10, thesampling clock generator 88 is implemented with an interpolator 90 and areceiver tracking unit (RTU) 92. The interpolator generates the samplingclock signal by interpolating between a number of phase clock signalsPCX (in this case four signals that are 90 degrees out of phase) inresponse to a tracking signal from the receiver tracking unit. Thereceiver tracking unit observes the data signal RX and adjusts thetracking signal so that the sampling clock signal causes the receiver tosample and redrive the data signal at an appropriate time. Thus, thesampling clock signal may dynamically track the data signal.

In one possible embodiment, the receiver tracking unit observestransitions in the data signal RX by over sampling the data signal andadjusting the sampling clock signal to sample and redrive the datasignal at the center of the data eye, i.e., at the midway point betweentransitions in the data signal. The sampling clock generator 88 mayinclude a loop filter that measures several bit cells and may eventuallydetermine that it should adjust the phase of the sampling clock signalto capture the data closer to the center of the data eye location. Theinput to the sampling clock generator may be taken from points otherthan the input of the receiver as shown in FIG. 10. For example, it maybe taken from the output of the receiver as well.

An embodiment of an I/O cell according to the inventive principles ofthis patent may be used with a scheme that trains the I/O cells todynamically track the data signal. For example, if the I/O cell of FIG.10 is used as one of the memory modules shown in FIG. 3, the host mayperiodically send training frames onto the outbound path. These trainingframes have an edge density that is adequate to assure that the receivertracking unit observes enough transitions in the data signal to be ableto adjust the sampling clock signal. Likewise, the outermost module inFIG. 3 may periodically send training frames onto the inbound path.

FIG. 11 illustrates another embodiment of an I/O cell according to theinventive principles of this patent. The embodiment of FIG. 11 issimilar to that of FIG. 9, but a buffer 94 is added in the data signalpath. The buffer 94 may be a jitter avoidance or drift compensationbuffer that compensates for voltage and temperature induced effects. Thebuffer resynchronizes the data signal TX to a transmit clock signal TC.The transmit clock signal is stable in the sense that its phase is notadjusted in response to the data signal the way the sample clock signalSC is.

In the embodiment of FIG. 11, the buffer is capable of operating ineither a pass-through mode, or a redrive mode in response to a modesignal. In pass-through mode, the signal passes through without beingsampled and redriven. In redrive mode, the signal is sampled andredriven in response to the clock signal. This enables the I/O cell tooperate in different redrive modes. In one possible embodiment, thebuffer operates in pass through mode if the mode signal is asserted.This is referred to as resample mode and may result in a shorter latencybecause the data signal is being redriven by the same clock that is usedto sample the data. When the mode signal is not asserted, the bufferoperates in redrive mode, so the data is resynchronized to the transmitclock. This is referred to as resync mode and may result in a longerlatency but may reduce jitter. The I/O cell may be designed into amemory buffer or module that has an input for receiving the mode signal.If the memory buffer or module is to be used on a system in which thereis a relatively short signal path to the next agent, the input may beasserted (or not asserted depending on polarity) to cause the I/O cellto operate in resample mode because more jitter may be tolerated on ashort signal path. On the other hand, if the memory buffer or module isto be used on a system in which there is a relatively long signal pathto the next agent, the input may be de-asserted to cause the I/O cell tooperate in resync mode because this reduces jitter, albeit at thepossible expense of longer latency. Alternatively, a registered flag maybe used on the memory buffer or module, or in a redrive circuit, or inthe I/O cell itself to control the mode signal.

FIG. 12 illustrates another embodiment of an I/O cell according to theinventive principles of this patent. In the embodiment of FIG. 12, thereceived and transmitted data signals RX and TX are differential signalsand are shown traversing the edge of an integrated circuit die on whichthe I/O cell may be fabricated. The receiver 86 includes a sampling unit96 and a termination unit 98. The sampling unit samples the incomingdata signal in response to a sampling clock signal SC which is generatedby interpolator 90 in response to phase clock signals from the samplingclock generator. The termination unit provides differential terminationand converts the differential data signal into a single-ended signal. Ajitter avoidance or drift compensation buffer 94 clocks data in responseto either the sampling clock signal SC or a stable transmit clock signalTC. A multiplexer 100 selectively couples data signals from either thebuffer 94 or a serializer 102 to a transmit latch 104. Read data signalsRDX[0 . . . n] are received at the I/O cell at serializer 102. Anothermultiplexer may be disposed between buffer 94 and transmit latch 104with one input connected to the buffer and another input connected to anoutput of the interpolator.

When the I/O cell needs to merge read data into the data stream, themultiplexer selects its input that is coupled to the serializer so thatthe transmit latch clocks the read data out of the I/O cell in responseto the transmit clock signal TC. Otherwise, the multiplexer selects thedata signal from the buffer which is then redriven by the transmitlatch. The transmit data signal is converted back to a differentialsignal by transmitter 106 before being driven onto the nextunidirectional link. Write data is taken from the output of the transmitlatch, collected in a deserializer 108, and then routed to a deskewcircuit, bit lane fail-over mechanism, or other circuitry. Thedeserializer may also provide a bit line clock signal BLC, which may bederived from the sample clock signal, to indicate when the write dataWDX[0 . . . n] is valid.

Some of the inventive principles of this patent relate to deskewingsignals separately from redrive paths. A redrive path is defined by oneor more components through which a signal propagates as it is receivedand redriven. For example, in the embodiments of FIGS. 9 and 10, theredrive path includes receiver 86. In the embodiment of FIG. 11, theredrive path includes receiver 86 and buffer 94. In the embodiment ofFIG. 12, the redrive path includes sampling unit 96, termination unit98, buffer 94, multiplexer 100, transmit latch 104, and transmitter 106.

According to some of the inventive principles of this patent, a deskewcircuit may be integrated into a redrive circuit such that theindividual bit lanes of the deskew circuit are included in the redrivepaths. Thus, the signals on the bit lanes may be deskewed in eachredrive circuit as it is redriven along a path. Alternatively, however,a deskew circuit according to the inventive principles of this patentmay be separate from the redrive paths. For example, in the embodimentof FIG. 7, a deskew circuit is shown separate not only from the redrivepaths in redrive circuit 60, but also from the entire redrive circuit.Alternatively, a deskew circuit according to the inventive principles ofthis patent may be integrated into the redrive circuit, but still beseparate from the redrive paths. For example, in the embodiment of FIG.12, one or more deskew latches may be located at the output ofserializer 102 and/or the input of deserializer 108.

The embodiments of methods and apparatus for deskewing signalsseparately from redrive paths as described above are exemplary only andare not limited to these specific examples. Moreover, the principlesrelating to deskewing signals separately from redrive paths according tothis patent are independent of other inventive principles of thispatent. For example, just as the embodiments of redrive circuitsillustrated in FIGS. 9-12 are not limited to use in memory systemshaving separate outbound and inbound paths, so too may the principlesrelating to deskewing signals separately from redrive paths according tothis patent may be used with other types of memory architecturesutilizing unidirectional links, e.g., an architecture that utilizes aring-type arrangement of links such as RamLink.

Some of the inventive principles of this patent relate to coping withfailed bit lanes. For example, any of the unidirectional links betweenany of the agents shown in the embodiments of FIGS. 3, 4, 5, 6 or 7 mayhave more than one bit lane. According to the inventive principles ofthis patent, one or more signals may be redirected on the bit lanes toavoid a bad bit lane. Any agent such as a memory controller (host),module, buffer, etc. may be capable of redirecting one or more signalson a number of bit lanes. A signal may be redirected at either or bothends of a link. Any agent may be capable of detecting a failed bit laneeither automatically or with assistance from another agent, and anyagent may be capable of redirecting signals responsive to a command fromanother agent.

FIG. 13 illustrates an embodiment of a fail-over circuit according tothe inventive principles of this patent. The fail-over circuit 110 ofFIG. 13 is shown along with an embodiment of a redrive circuit 112 forpurposes of illustration only, but the inventive principles are notlimited to use with any particular redrive circuit, nor is the fail-overcircuit limited to the specific details shown in FIG. 13. Redrivecircuit 112 includes a number of bit lanes arranged to receive andredrive signals on unidirectional links. Each bit lane is embodied as anI/O cell 114 having a receiver 116 and a transmitter 118.

A fail-over circuit refers to a circuit that is capable of redirectingone or more signals to or from a plurality of bit lanes. In theembodiment of FIG. 13, the fail-over circuit is implemented as amultiplexer having one or more multiplexer switches 120. Each switch hasa first input coupled to one bit lane and a second input coupled to anadjacent bit lane so that it may redirect signals from either bit laneto its output. The embodiment shown in FIG. 13 is shown with sixswitches to service six bit lanes, but any number switches and bit lanesmay be used, and the switches may be arranged in various configurationsother than the adjacent bit lane configuration as shown.

During a normal mode of operation, each of the switches directs thesignal from its first input to its output as shown in FIG. 14 so thatwrite data signals WD0, WD1, WD2, WD3, WD4, and WD5 are directed tooutputs OUT0, OUT1, OUT2, OUT3, OUT4, and OUT5, respectively. In such anembodiment, one of the bit lanes, for example, the bit lanecorresponding to WD5, may be used for error checking the data on theother bit lanes.

If a bad bit lane is detected, the multiplexer may operate in afail-over mode in which one or more of the switches are manipulated tomap out the bad bit lane. For example, if the bit lane associated withWD3 does not operate properly, the multiplexer switches may redirectwrite data signals WD4 and WD5 to outputs OUT3 and OUT4, respectively asshown in FIG. 15. In this mode, one bit lane worth of signal capacity islost. If one of the bit lanes had been designated for error checking,signals originally intended for the bad bit lane may be rerouted overthe error checking lane, and the error checking function may bedisabled.

The outputs of the fail-over circuit may be coupled to a memoryinterface, to a memory device, or to other circuitry. In the embodimentof FIG. 13, the fail-over circuit is shown separate from the redrivecircuit, but it may also be integrated into the redrive circuit. Afail-over circuit according to the inventive principles of this patentmay be realized with simple multiplexers as shown, but otherarrangements such as a full crossbar switch are also possible.

The embodiment of a fail-over circuit shown in FIG. 13 is arranged tocouple write data from the bit lanes to its outputs. Alternatively, anembodiment of a fail-over circuit according to the inventive principlesof this patent may be arranged to transfer data in the oppositedirection, in which case the outputs OUTX would become inputs thatreceive read data, the multiplexer switches may be referred to asdemultiplexer switches, and each of the I/O cells may have a multiplexerbetween the receiver and transmitter to merge the read data from thefail-over circuit into the bit lane. Thus a multiplexer refers to both amultiplexer and a demultiplexer. As another alternative, an embodimentof a fail-over circuit according to the inventive principles of thispatent may be arranged for bi-directional data flow between the bitlanes and memory device, memory interface, or other circuitry.

A memory buffer, memory module, memory controller (host), or other agenthaving bit lane fail-over capability may also have various capabilitiesfor detecting failed bit lanes, redirecting signals, mapping out bad bitlanes, and the like according to the inventive principles of thispatent. For example, an agent having the embodiment of a fail-overcircuit shown in FIG. 13 may be designed so that it can detect a failedbit lane, e.g., by observing a test data pattern sent by another agent,and redirecting signals to map-out the failed bit lane. Alternatively,the agent may be designed so that it may map out a failed bit lane inresponse to a command from another agent, for example, a memorycontroller that instructs one or more agents on a memory channel.Alternatively, the agent may have both capabilities.

FIG. 16 illustrates an embodiment of a memory buffer with bit lanefail-over capability according to the inventive principles of thispatent. The embodiment of FIG. 16 is similar to that of FIG. 7 but alsoincludes a fail-over circuit 122 coupled between the deskew circuit 70and the memory interface 66. Alternative embodiments are possible. Forexample, the fail-over circuit may be disposed between the redrivecircuit 60 and the deskew circuit, or it may be integrated into theredrive circuit. The embodiment of FIG. 16 also includes anotherfail-over circuit 124 which is shown coupled between the multiplexer 74and redrive circuit 62, but which may also be integrated into theredrive circuit or arranged in other ways. The memory buffer of FIG. 16may alternatively be embodied as a memory module, in which case thememory interface is replaced by a memory device.

FIG. 17 illustrates an embodiment of a memory controller with bit lanefail-over capability according to the inventive principles of thispatent. The controller of FIG. 17 includes outbound and inboundunidirectional link interfaces 126 and 128 having a plurality of bitslanes which, in this embodiment, include a number of transmitters and anumber of receivers, respectively. Fail-over circuits 130 and 132 arecoupled to the bit lanes in the interfaces 126 and 128, respectively. Inthe embodiment of FIG. 17, the fail-over circuits are shown separatefrom the link interfaces, but they may alternatively be integral withthe interfaces. The controller may be capable of detecting a failed bitlane, in which case the fail-over circuits may map out the failed bitlane. Additionally or alternatively, the controller may be capable ofissuing a command that directs an agent to map out a failed bit lane.

Additional fail-over methods and apparatus according to the inventiveprinciples of this patent will now be described in the context of anexemplary embodiment of a complete memory channel system includingadditional embodiments of a memory controller (host), memory modules,and memory buffers according to the inventive principles of this patent.None of the components, however, are limited to this exemplary system orany of the details described therein.

The exemplary system includes an embodiment of a host having fail-overcapabilities such as those described with reference to FIG. 17 andembodiments of one or more memory modules having buffers with fail-overcapabilities such as those described with reference to FIG. 16. In thisexample, the host and modules are arranged in a channel configurationhaving outbound and inbound paths such as that shown in FIG. 7, althoughthe system may only include one module.

In this example, the host and modules are interconnected with a systemmanagement bus known as “SMBus”, which is a serial bus system used tomanage components in a system. However, the use of SMBus is notnecessary to the inventive principles of this patent, and other forms ofcommunication between components may be used, including the memorychannel paths themselves.

An embodiment of a method according to the inventive principles of thispatent for detecting and mapping out a failed bit lane in the exemplarysystem proceeds as follows. The host transmits a test pattern on eachbit lane of the outbound path. The test pattern is received and redrivenalong the outbound path by the buffer on each module until it reachesthe outermost module. The outermost module then transmits a test patternon each bit lane of the inbound path. The test pattern is received andredriven along the inbound path by the buffer on each module until itreaches the host. The host and the buffers on the modules observe thetest pattern on each bit lane of the inbound and/or outbound paths tocheck for proper bit lane operation. The bit lanes in the inbound andoutbound paths may be tested concurrently.

Failed bit lanes are reported by sending results to the host through theSMBus and/or by transmitting a results frame over the channel to thehost. Such a results frame may be initiated on the inbound path by theoutermost module, and the other modules, if any, may merge their resultsinformation into the data in the inbound path. If the results from eachmodule are transmitted redundantly on more than one bit lane, a failedbit lane is unlikely to interfere with reporting the results.

Once the host receives the results, it may issue a configuration commandto the modules, through the SMBus, over the channel, or through anyother form of communication. The configuration command instructs themodules which, if any, bit lanes are bad and should be mapped out. Themodules respond to the configuration command by manipulating one or morefail-over circuits to redirect signals around bad bit lanes, if any, andreconfiguring any internal functionality to accommodate the loss of abit lane. For example, if one bit lane was designated for error checkingdata, the buffer or module may disable error checking functions.

The embodiments of fail-over methods and apparatus described above areexemplary only, and the inventive principles of this patent are notlimited to these specific examples. The principles of fail-over methodsand apparatus according to this patent have been described withreference to a memory system having separate inbound and outbound pathssuch as the embodiment of FIG. 3, but the principles may also be appliedto any memory architecture utilizing unidirectional links, for examplean architecture that utilizes a ring-type arrangement of links such asRamLink.

Some of the inventive principles of this patent relate to permutingstatus patterns. In memory systems such as those described above withreference to FIGS. 1 and 3 where memory read and write data istransferred between memory agents, it may also be useful to send statusinformation such as idle patterns, alert patterns, and other statusinformation between memory agents. This may be accomplished by sendingdata patterns and status patterns on the same link or links that connectthe memory agents. According to the inventive principles of this patent,the status patterns may be permuted over time.

For example, referring to FIG. 3, the memory controller 50 may sendframes having data patterns such a read commands to one or more of themodules 52 which respond by sending frames having data patterns such asread data back to the controller. It may be useful for the one or moreof the modules to send a frame having an idle pattern back to the memorycontroller, for example, if the module was not able to retrieve readdata from a memory device 58 fast enough. A predetermined data patternmay be designated as an idle pattern so that, if the memory controllerreceives the idle pattern, it knows it is not receiving read data. Thismay cause problems, however, if the actual read data pattern happens tomatch the designated idle pattern.

According to the inventive principles of this patent, the memorycontroller and one or more modules may both be capable of permuting theidle pattern in a predictable manner so that the idle pattern changesover time. For example, the memory controller and modules may change theidle pattern according to a predetermined sequence each time an idleframe is sent and/or received. An embodiment of such a method accordingto the inventive principles of this patent is illustrated in FIG. 18.Thus, if the memory controller sends a read command frame (158) andreceives a response frame (160) having the current idle pattern (162),it may resend the same read command (164). If the second response frame(166) contains the same pattern as the first (168), it interprets thepattern as actual read data (170). If, however, the pattern in thesecond response frame matches the permuted idle pattern (168), thememory controller knows that the first response frame was an idle frame(172).

According to the inventive principles of this patent, the statusinformation sent in status patterns may be idle patterns, alertpatterns, and other status information such as command error informationfrom a module, thermal overload information from a module, andinformation that indicates that a module has detected the presence ofanother module on the outbound path of memory channel. Some types ofstatus patterns may be implemented as complementary patterns. Forexample, an alert pattern (which may be used to notify an agent of anerror condition) may be implemented as the logical complement of an idlepattern. This may simplify the implementation by, for example, allowinga memory agent to use the same pattern generator for idle and alertpatters. The use of complementary status patterns may be beneficial evenif permuting patterns are not used.

A memory agent according to the inventive principles of this patent mayalso be capable of intentionally generating an error such as a cyclicalredundancy check (CRC) error in a status pattern. Such a technique maybe useful as an alternative or supplemental way to distinguish a datapattern from a status pattern. For example, in some memory systems, eachframe is sent along with a CRC code that used to check the integrity ofthe data in the frame. According to the inventive principles of thispatent, a memory agent may intentionally send the wrong CRC code withframe that contains a status pattern. The receiving agent may theninterpret the frame as a status frame rather than a data frame. Somememory systems may utilize a path or paths having an extra bit lane tocarry CRC data. If such a system is capable of operating in a fail-overmode, the agent or agents may only utilize an intentional CRC error ifnot operating in fail-over mode. As used herein, the term CRC refers notonly to a cyclical redundancy check, but also to any other type of errorchecking scheme used to verify the integrity of a frame or pattern.

Although the principles of status pattern permuting and handlingaccording to the inventive principles of this patent are applicable toany type of memory agent, and are independent of other inventiveprinciples of this patent, some additional aspects will be describedwith respect to a memory buffer such as the embodiment shown in FIG. 7and in the context of a system such as the embodiment shown in FIG. 6.Referring to FIG. 6, if the memory buffer 64 is the outermost agent on amemory channel, it may be capable of constantly transmitting permutingidle status frames on the inbound link 56B whenever it is not sendingdata that the host has requested from any memory devices attached to thememory interface 66.

FIG. 19 illustrates an embodiment of a permuting pattern generator inaccordance with the inventive principles of this patent. The embodimentof FIG. 19 is a 12-bit linear-feedback shift register (LFSR) with apolynomial of x¹²+x⁷+x⁴+x³+1. The initial state may be set to000000000001, and the LFSR cycles through 2¹²−1 states (4095 frames)before the pattern is repeated. Each bit of the LFSR may be mapped to abit lane in a link on a data path, and each bit may be used for all ofthe transfers that occur on the corresponding bit lane during an entireframe. For example, in a system having a data path with 12 bit lanes ineach link, the output from each stage of the LFSR may be mapped to oneof the bit lanes. Additional lanes, for example, a 13th bit lane, may beaccommodated by utilizing the value from the least significant bit ofthe LFSR delayed by one frame.

FIG. 20 illustrates an example of the first status pattern generated bythe permuting pattern generator of FIG. 19. In this example, a frame is12 transfers long. FIGS. 21-22 illustrate the second, third and forthstatus patterns, respectively. By using the same value on each bit laneduring an entire frame, electromagnetic interference (EMI or noise) maybe reduced.

The 13 bit lane by 12 bit transfer frame illustrated here is by way ofexample, and the inventive principles of this patent are not limited tothese details, nor to the specific embodiment of a permuting patterngenerator described above. For example, a permuting pattern generatoraccording to the inventive principles of this patent need not beimplemented with dedicated logic circuitry such as the LFSR describedabove. Alternatively it may be implemented with programmable logic, oras an algorithm in a processor or other programmable state machine thatmay be used to oversee and/or implement the logic in the memoryinterface or other functionality of a buffer or other memory agent thatutilizes permuting status patterns.

Some additional inventive principles of this patent relate to utilizingmore than one bit lane to detect the presence of a memory agent on amemory link. For example, in the embodiment of a memory buffer shown inFIG. 7, the buffer may be capable of detecting whether there is anothermemory agent coupled to the outbound link 54B. This may be accomplishedby utilizing a single bit lane in the link to test for the presence ofanother memory agent. If there is more than one bit lane in the link,however, more than one of the bit lanes may be used to detect thepresence of another memory agent according to the inventive principlesof this patent. This may prevent the existence of a bad bit lane frominterfering with the presence detect operation.

For convenience, the inventive principles of this patent relating toutilizing more than one bit lane to detect the presence of a memoryagent will be referred to individually and collectively as redundantpresence detect. Redundant presence detect may be applied to any type ofmemory agent having a link interface with a plurality of bit lanes. Forexample, any two or more of the transmitters 118 shown in the embodimentof FIG. 13 may be considered a link interface, in this case a transmitlink interface. Likewise, any two or more of the receivers 116 shown inFIG. 13 may be considered a link interface, in this case a receive linkinterface. Redundant presence detect may be applied to either of theselink interfaces, as well as either of the link interfaces 126 and 128shown in the embodiment of FIG. 17.

Returning to the embodiment of FIG. 7 as an example again, the memorybuffer may drive three bit lanes on its inbound transmit link 56B to apredetermined presence detect logic level, e.g., one, to signal itspresence to another buffer when a presence detect event such as a resetoccurs. Also during a presence detect event, a second such memory bufferlocated inbound from the first buffer on a channel may configure thecorresponding three bit lanes on its inbound receive link 56A to detectthe presence of the first buffer. In this example, the first memorybuffer will be referred to as an outer agent, and the second buffer willbe referred to as an inner agent.

An example of a technique for configuring a bit lane to detect thepresence of another agent is to have the receiver for that bit lane tryto place a bias current on the bit lane so as to force the bit lane tothe opposite of the presence detect logic level. If another memory agentis coupled to the bit lane during a presence detect event, itstransmitter on that bit lane will force the bit lane to the presencedetect logic level.

If the inner agent detects the presence detect logic level on two of thethree bit lanes, it knows that the outer agent is present and it mayleave all or a portion of its outer port enabled. (In this example, theouter port includes the link interface for the outbound link 54B and thelink interface for the inbound link 56A.) If the inner agent fails todetect the presence detect logic level on at least two of the three bitlanes, it may decide that an outer agent is not present and it maydisable all or a portion of its outer port. The inner agent may becapable of reporting the presence or absence of an outer agent toanother agent, for example to a memory controller in response to astatus check command.

An agent utilizing redundant presence detect may also be capable ofsignaling a presence detect event to another agent. For example, if areset event is communicated to the buffer of FIG. 7 through a resetcommand on the outbound path, this command may be relayed to an outeragent, if any, by redrive circuit 60. This may place both agents in apresence detect mode.

Redundant presence detect according to the inventive principles of thispatent is not limited to the specific embodiments discussed above. Forexample, only two bit lanes may be used for presence detect instead ofthree as in the example above, in which case the inner agent would onlyneed to detect the presence detect logic level on a single bit lane toconclude that an outer agent was present. Likewise, redundant presencedetect may be applied to systems and components utilizing various othertypes of memory architectures, e.g., an architecture that utilizes aring-type arrangement of links such as RamLink.

Some additional inventive principles according to this patent relate tohot insertion and/or removal of components from a memory channel—thatis, adding and/or removing components while the memory channel isoperating. FIG. 24 illustrates an embodiment of a memory agent 134according to the inventive principles of this patent. The embodiment ofFIG. 24 may be a memory module, memory buffer, memory controller, etc.The agent includes a first port 136 and a second port 138. If the agentis assumed, for purposes of illustration only, to be a memory modulesuch as one of modules 52 in the embodiment of FIG. 6, the first portmay be designated as an inner port since it may be arranged tocommunicate with other agents on the memory channel that are locatedcloser to the memory controller. Likewise, the second port may bedesignated as an outer port since it may be arranged to communicatedwith agents on the memory channel that are located further away from thememory controller. These designations are for purposes of illustrationonly, and the inventive principles are not limited to these details ofthe memory agent nor to the particulars of the memory channel shown inFIG. 6. These principles may also be applicable to other memory channelarchitectures such as the RamLink architecture shown in FIG. 1.

Each port of a memory agent according to the inventive principles ofthis patent has one or more link interfaces. In the embodiment of FIG.24, each port has both a receive link interface and a transmit linkinterface. The inner port 136 has a receive link interface 140 which maybe one or more receivers that are part of a redrive circuit 60, and atransmit link interface 142 which may be one or more transmitters thatare part of another redrive circuit 62. The outer port has receive andtransmit link interfaces 144 and 146, respectively, which are also partof redrive circuits 62 and 60, respectively. Link interfaces 140 and 146may be coupled to outbound links 54A and 54B, respectively, and linkinterfaces 142 and 144 may by coupled to inbound links 56B and 56A,respectively. Each of the link interfaces may have one or more bitlanes, and the bit lanes and interfaces may be referred to using anycombination of this terminology. For example, the bit lanes in interface142 may be referred to as the inbound transmit or inbound Tx bit lanes.The bit lanes in interface 144 may be referred to as the inbound receiveor inbound Rx bit lanes.

The embodiment of FIG. 24 is exemplary only, and memory agents and portsmay be embodied in different ways. For example, link interfaces are notnecessarily part of a redrive circuit. This is illustrated in theembodiment of a memory controller shown in FIG. 17 wherein a port mayinclude the link interfaces 126 and 128 which are not part of redrivecircuits. The link interfaces may include only one or any number of bitlanes, and a port may only have a receive link interface or a transmitinterface.

A memory agent according to the inventive principles of this patent maybe capable of detecting the presence of another memory agent on one ofits ports, and it may be capable of taking various actions depending onthe presence or absence of another memory agent. For example, the memoryagent of FIG. 24 may be capable of disabling all or a portion of itsouter port if another memory agent is not present at the port. It may becapable of reporting the presence or absence of an outer agent toanother agent, for example to a memory controller through its innerport. The memory agent of FIG. 24 may be capable of performing apresence detect operation which may include signaling a presence detectevent to a potential outer agent on the outer port. It may also becapable of performing a fast reset operation.

Some additional inventive principles which may facilitate hotadd/removal in accordance with this patent application will be describedin the context of an example embodiment of a memory system. The exampleembodiment will be described with reference to the memory agent of FIG.24 in the context of a memory system such as the embodiment of FIG. 6.In this example embodiment, it will be assumed that the memory agent ofFIG. 24 is used to embody one or more of the buffers in FIG. 6, which inturn are part of modules having memory devices. All of these details,however, are for purposes of explanation only, and the inventiveprinciples are not limited to these details.

In the example system, the memory agents may be capable of executingfast reset operations, full reset operations, and/or various polling orpresence detect operations. In the example system, a minimum number ofclock transitions may be necessary to keep the derived clocks on eachbit lane locked to the data stream. Thus, the memory controller (orhost) may initiate a reset operation by sending a continuous stream ofones or zeros on one or more of the bit lanes in the outbound path for apredetermined period of time. Since the data is redriven by each bufferon the path, all of the buffers receive the reset command, or event. Inthe example system, the three least significant bit (LSB) lanes may beused to signal a reset operation. The receiving agent may detect thereset event by sensing the stream of zeros or ones on any two of thethree LSBs. This may assure that the presence of one failed bit lanedoes not interfere with a reset operation, but the inventive principles,which do not even require more than one bit lane, are not limited tosuch an implementation.

In the example system, the host may send a continuous stream of zeroshold all of the agents on the channel (in this example modules havingbuffers) in a first reset state indefinitely, for example while the hostis held in reset by external conditions. The host may then send a streamof ones for a first amount of time, e.g., two frame periods, and thenback to zeros to signal the other agents to execute a fast resetoperation. Alternatively, the host may send a stream of ones for asecond amount of time, e.g., more than two frame periods, to signal theother buffers to execute a full reset operation. A full reset mayinclude various internal calibration operations such as impedancematching on the links, current source calibration in any receiver ordrive circuitry, receiver offset cancellation, and the like. After thecalibration operations are performed, the host may then signal thebuffers to transition to the fast reset operation.

A fast reset operation may bypass certain operations performed during afull reset such as calibration operations. A fast reset operation maybegin with a presence detect operation. During a presence detectoperation, each buffer on the channel may place a current on the threeLSB inbound Rx bit lanes to force the bits to zero if they are notconnected to an outer agent. Also during a presence detect operation,each buffer may drive the three LSB inbound Tx bit lanes to one. Eachbuffer may then check its three LSB inbound Rx bit lanes, and if itdetects ones on two of the three lanes, it may leave its outer portenabled and update a status register accordingly. If the buffer does notdetect two ones, it may assume that there is no outer agent, disable allor a portion of its outer port, configure itself to perform thefunctions of the outermost agent on the channel, and/or update a statusregister accordingly. A host may follow a similar presence detectoperation to determine if any agents are on the channel. The buffers mayrelay the status information to the host in status frames in response tostatus requests from the host.

After a presence detect operation, the buffers in the example system maytransition through various other operations during a fast reset such asa clock training state to train the local clocks on the buffers to lockonto the data stream, a frame training state to align frames that aresent over the channel, bit lane tests to check the operation of all bitlanes and place the buffers in fail-over mode if they have such acapability, etc. Also, once the host knows how many other agents areconnected to the channel, it may adjust the frame size, timing, etc. toaccommodate all of the agents.

In the example system, the memory agents may also or alternatively becapable of performing various polling operations to detect the presenceof newly added agents on the channel. For example, each buffer may becapable of performing a polling operation on its outer port if it is theoutermost buffer to determine if a new agent has been added to thechannel. FIG. 25 illustrates an embodiment of such a polling operationaccording to the inventive principles of the patent.

At 148, the agent may disable all or a portion of its outer port. If theagent is a buffer or module, it may wait for a poll command from thehost to transition to a hot reset operation at 150. If the agent is ahost, it may disable all or a portion of its outer port and wait for awake up command from a system environment. Upon receiving the wake upcommand, it may turn enable all or a portion of its outer port andtransition to a reset state.

At 150, the agent may enable its outer port and drive zeros onto thethree LSB outbound Tx bit lanes to send a reset to a potential new agenton its outer port. The agent may then transition to a hot calibrationoperation at 152.

At 152, the agent may drive ones onto the three LSB outbound Tx bitlanes to force a potential new agent through a full reset includingcalibration operations, since a newly detected agent would presumablyneed to be calibrated. The agent may then transition to a hot detectoperation at 154.

At 154, the agent may drive zeros onto the three LSB outbound Tx bitlanes and place a bias current on the three LSB inbound Rx bit lanes toforce the bits to zero of they are not connected to an outer agent. Theagent may then check the three LSB inbound Rx bit lanes, and if itdetects at least two ones, it may decide at 155 that an outer agent ispresent and transition to a hot agent present operation at 156.Otherwise, the agent may decide at 155 that an outer agent is notpresent and transition back to the sleep operation at 148.

At 156, the agent may update a status register to indicate that it hasdetected an outer agent and then relay this information to the host, forexample, in response to a status request, or take some other action torelay the information to the host or other agent. The agent may alsowait to receive a channel reset.

The host may become aware of the newly detected agent, either throughperiodic status requests, or other techniques and initiate a fast resetto re-initialize the entire channel with the new agent on the channeland accommodated in the channel timing.

The following are some additional embodiments of hot add/removalsequences according to the inventive principles of this patent. Theseadditional embodiments are also described with reference to theembodiment of the memory system shown in FIG. 6 in the context of alarger system, such as a server having a user interface and systemfirmware, that employs the memory system of FIG. 6. The inventiveprinciples illustrated by these additional embodiments, however, are notlimited to the specific details described herein.

A hot add sequence according to the inventive principles of this patentmay begin when a user appends a new agent onto the memory channel, forexample on the outer port of the outermost agent. The user may informthe system firmware that an agent has been appended. The firmware maythen cause power to be applied to the appended agent and inform the hostthrough a wake up command that an agent has been appended. The host maythen send a poll command to the previous outermost agent, which then maycycle through a polling operation such as the one described above withreference to FIG. 25. After the polling operation, the previousoutermost agent may report the presence of a new outer agent. The hostmay then detect the presence of the new agent and issue a fast resetcommand to bring the new agent into operation and retime the entirechannel. After the new agent is operational, the host may interrupt thesystem firmware to report that the new agent is operational.Alternatively, the host may wait for the system firmware to query thehost to determine if the new agent is operational. The system firmwaremay then configure the host to accommodate any new hardware presented bythe new agents such as new memory devices that may be present if theagent was a memory module or buffer.

A hot removal sequence according to the inventive principles of thispatent may begin when a user informs the system that a specific agent ona memory channel is to be removed. The system may remove a correspondinghost address range from a system map. If the system uses mirroring, thesystem may remap the host address ranges to agent mirrors. The systemmay then copy or move data from the host address range to otherlocations if not already mirrored. The system may then poll until alloutstanding transactions are completed. The system may then cause thehost to send a command to the agent just inside of the agent to beremoved that causes this agent to assume it is the outermost agent onthe channel, thereby causing it to disable its outer port and assume thefunctions of the outermost agent during subsequent fast resets. (A fullreset would override this command.) The system may then initiate a fastreset to shut down the selected agent and any channel interfaces forcomponents attached to the selected agent. The system may thendisconnect power to the selected agent and notify the user that theagent may be removed.

A hot replace sequence according to the inventive principles of thispatent may begin when the hot remove sequence described above iscompleted. The user may add a new agent in place of the one removed andthen inform the system firmware that the new agent has been added. Therunning system may then prepare the host for the newly replacedcomponent and supply power to the new component. System firmware maythen cause the host to send a command to the previous outermost agent tolet is know that it should no longer assume that it is the outermostagent. This may cause the previous outermost agent to enable its outerport in response to the next reset, and wait for a poll command.Firmware may then instruct the host to send a poll command to theprevious outermost agent which may then perform a polling operation suchas the one described above with reference to FIG. 25, therebyinitializing the new agent. The previous outermost agent may then reportthe presence of a new outer agent. The host may then detect the presenceof the new agent and issue a fast reset command to bring the new agentinto operation and retime the entire channel. After the new agent isoperational, the host may interrupt the system firmware to report thatthe new agent is operational. Alternatively, the host may wait for thesystem firmware to query the host to determine if the new agent isoperational.

Some of the inventive principles of this patent relate to accumulatingdata between a data path and a memory device. FIG. 26 illustrates anembodiment of a memory module utilizing data accumulation according tothe inventive principles of this patent. The module 174 of FIG. 26includes one or more memory devices 180 and a redrive circuit 176 toreceive one or more signals on point-to-point link 178A, and thenredrive the signals on point-to-point link 178B. A data accumulator 182is disposed between the redrive circuit and a memory device. An optionalsecond redrive circuit 184 is arranged to receive one or more signals onpoint-to-point link 186A and redrive the signals on point-to-point link186B. In this embodiment, the point-to-point links are shown asunidirectional links, but the inventive principles are not limited tounidirectional links.

The data accumulator 182 may be a first-in, first-out (FIFO) datastructure or any other type of suitable queue or buffer. The use of adata accumulator may allow for bandwidth mismatches. For example, amemory device having a high-bandwidth burst mode may be used for thememory device 180. The bandwidth of the data path formed from theunidirectional links may be less than the burst mode of the memorydevice in order to reduce pin count, power consumption, andmanufacturing and operating costs. The memory device, however, may needto receive data at full bandwidth for proper operation in burst mode. Byutilizing a data accumulator, write data from the data path may beaccumulated at a rate less than the burst rate of the memory device, andthen delivered to the memory device at its full burst rate.

The module of FIG. 26 is not limited to any particular arrangement ofunidirectional links or any particular arrangement for transferring datato and/or from the redrive circuits. Data is shown flowing from theredrive circuit 176 to the one or more memory devices 180 through dataaccumulator 182, but the direction could be reversed, and additionaldata accumulators may be added between the redrive circuit and memorydevices. Likewise, if the optional second redrive circuit 184 isincluded, data may flow may be either to or from a memory device. One ormore data accumulators may be included between the second redrivecircuit and the memory devices to accumulate write data to a memorydevice and/or read data from a memory device.

FIG. 27 illustrates another embodiment of a memory module and anembodiment of a memory buffer utilizing data accumulation according tothe inventive principles of this patent. The module 174 of FIG. 27includes a memory buffer 188 having two redrive circuits 176 and 184,and a memory interface 190 arranged to transfer data to and from one ormore memory devices 180. As with the module of FIG. 26, only one of theredrive circuits is needed in the buffer of FIG. 27. The memoryinterface 190 includes a data accumulator 182 which may be arranged toaccumulate data as it flows between the redrive circuit 176 and memorydevices 180. Data flow may be in either direction, and an additionalaccumulator may be included to accommodate data accumulation in bothdirections. Likewise, if the second redrive circuit 184 is included, oneor more additional data accumulators maybe included to interface withthe second redrive circuit.

FIG. 28 illustrates another example embodiment of a memory bufferutilizing data accumulation according to the inventive principles ofthis patent. The buffer 64 of FIG. 28 is similar to that of FIG. 7, butthe memory interface 66 includes a FIFO-type write data accumulator 192and a FIFO-type read data accumulator 194. Some possible implementationdetails will now be described with reference the embodiment of FIG. 28,but the inventive principles of this patent are not limited to thesedetails.

Write data from the first redrive circuit 60 is accumulated in the writeFIFO at whatever data rate the outbound path happens to be operating at.Once enough write data is accumulated, it may be written to one or morememory devices at full burst rate through memory bus 68. The read FIFO194 may perform data capture from the memory device at full burst rate,and levelize the data prior to transferring the read data to the secondredrive circuit 62 through multiplexer 74.

The write FIFO may be constructed so that it can accumulate multiplebursts of data prior to bursting the data to a memory device. Thisallows the read-write-read memory bus turn around penalty to beamortized over a number of write operations. The write FIFO may also beconstructed so that additional data may be loaded into the FIFO whiledata is being delivered to the memory device. This allows the depth ofthe FIFO to be smaller than the number of transfers in a burst. As afurther refinement, a data pre-accumulator may be located ahead of thewrite FIFO and set up to speculatively capture write data from the datapath without regard as to whether the data is intended for thisparticular memory buffer 64. Once the target memory buffer isidentified, the data in the pre-accumulator may be transferred to thewrite FIFO if this memory buffer was the intended recipient, otherwiseit may simply be discarded.

As with the other memory modules and/or buffers disclosed in thispatent, the embodiments of memory modules and buffers described withreference to FIGS. 26-28 are not limited to any particular mechanicalarrangements, and may be used to interface devices other than memorydevices to a channel.

Some additional inventive principles of this patent application relateto transmitting frames with early delivery of a CRC code for a portionof the frame. FIG. 29 illustrates an example embodiment of a frameaccording to the inventive principles of this patent. In this example,the frame is 10 bits wide and 12 transfers long, but any other width andlength may be used. The bits in the row identified as transfer “0” aretransferred first, followed by the bits in the row identified astransfer “1”, etc. The frame is completed when the bits in row “11” aretransferred. The first four rows (shown in shading) are designatedarbitrarily as a first portion of the frame, while the remainder of theframe is designated as a second portion of the frame.

In prior art frame transfer schemes, a CRC code for error checking theentire frame is typically placed at the end of the frame. According tothe inventive principles of this patent, a CRC code for the firstportion of a frame may be transferred before completing, or preferablyeven beginning, the transfer of the second portion. This early deliveryof a CRC allows the memory agent receiving the frame to error check thefirst portion of the frame, and preferably begin utilizing anyinformation contained therein, before the second portion of the frame iscompleted.

For example, if this technique is used with a memory agent havingapparatus that buffers memory devices such as DRAM chips from acommunication channel, a DRAM command may be placed in the first portionof the frame, while a data payload may be placed in the second portion.Early delivery of the CRC for the first portion allows apparatus in thememory agent receiving the frame to error check the command in the firstportion of the frame and forward it to a DRAM chip before the datapayload in the second portion is received.

The CRC code for the first portion may be included as part of the firstportion of the frame. It may be placed at the end of the first portion,distributed throughout the first portion, contained only partly in thefirst portion, or transferred in any other suitable manner such that itis received before the end of the second portion. The CRC code for thefirst portion may be combined with other CRC codes to create compoundcodes, or may be the result of compounding with other codes both withinand outside of the frame.

As used herein, the first portion need not be literally first in theframe, but may also be any portion that is received earlier than a laterportion. Likewise, the second portion may be the next portion after thefirst, but there may also be other portions between the first and secondportions or after the second portion, and the first and second portionsmight even overlap, so long as the effect is that a CRC for the firstportion may be transferred early so that error checking of the firstportion may begin before the frame is completely transferred.

A second CRC code for the second portion of the frame may be placed atthe end of the second portion, distributed throughout the secondportion, contained only party in the second portion, or transferred inany other suitable manner. The second CRC code may cover only the secondportion of the frame, may cover the entire frame, or may by compoundedwith other CRC codes in other ways.

CRC refers not only to cyclical redundancy checking, but also to anyother type of error checking scheme used to verify the integrity of aframe.

Some additional inventive principles of this patent application relateto organizing CRC codes across multiple frames. FIG. 30 illustrates anexample embodiment of a scheme for delivering a CRC code across multipleframes according to the inventive principles of this patent. In thisexample, a frame is 10 bits wide and 12 transfers long, but any otherwidth and length may be used. The bits in the row identified as transfer“0” are transferred first, followed by the bits in the row identified astransfer “1”, etc. The frame is completed when the bits in row “11” aretransferred. The frames in FIG. 30 are transferred sequentially withframe N-1 being transferred first, followed by frame N.

A portion of a CRC code for frame N-1 is shown shaded in frame N-1arbitrarily in the position of bit “9” in the rows identified astransfers “4” through “11” of frame N-1. Another portion of the CRC codefor frame N-1 is shown shaded in frame N arbitrarily in bits “9” through“7” in rows “0” through “1” and in bits “9” through “6” in rows “2”through “3”.

The CRC code for frame N-1 (which is shown distributed over frames N-1and N) may be intended for error checking all of frame N-1, just aportion of frame N-1, or all or a portion of some other frame. MultipleCRC codes (or portions of CRC codes) for different portions of frame N-1may also be contained in the same frame or combined with CRC codes (orportions of CRC codes) from other frames to create compound CRC codes.

The inventive principles of this patent application relating toorganizing CRC codes across multiple frames are independent of thoseinventive principles relating to early delivery of CRC codes. Thesedifferent principles may be combined, however, thereby giving rise toadditional inventive principles. FIG. 31 illustrates an embodiment of aframe transfer scheme incorporating early CRC delivery and distributedCRC codes according to various inventive principles of this patent.Numerous implementation details will be described by way of example, butthe inventive principles are not limited to these details.

In the example embodiment of FIG. 31, each frame is again 10 bits wideand 12 transfers long, and frame N-1 is transferred first, followed byframe N. The first four rows of Frame N-1 are once again arbitrarilydesignated as a first portion and will be referred to as the commandportion. The last four rows are arbitrarily designated as a secondportion and will be referred to as the data portion.

The command portion includes 24 bits of command information in theaC[23:0] field, two additional bits of command information or frame typeencoding in the F[1:0] field, and a 14-bit compound CRC checksum in theaE[13:0] field. The aC[23:0] field and the F[1:0] field will be referredto collectively as the “A” command. The aE[13:0] field provides errordetection coverage across the F[1:0], aC[23:0] and aE[13:0] fields.

The data portion includes 72 bits of data in the B[71:0] field which maybe used for additional commands, command extensions, data transfer,etc., and a portion of a 22-bit compound CRC checksum identified asFE[21:0]. Eight of the 22 bits are located in the FE[21:14] field inframe N-1 (the current frame). The other 14 bits are located in theFE[13:0] field which is shown outside of the frame in FIG. 31 becausethis portion of the checksum will be latched and compounded with a CRCcode in frame N (the next frame).

To generate a frame at the transmitting memory agent, a 22-bit CRC(referred to as CRC22[21:0] or the “data CRC”) is generated from the72-bit data B[71:0]. A 14-bit CRC (referred to as CRC14[13:0] or the“command CRC”) is generated from the 26-bit “A” command F[1:0]aC[23:0].Eight bits of the 22-bit data CRC are used directly as FE[21:14] and arelocated in the 10th bit lane (bit lane “9”) of the current frame. Theremaining 14 bits of the 22-bit data CRC become FE[13:0] and arecombined with the 14-bit CRC generated from the 26-bit “A” command inthe next frame using a bit-wise exclusive-or (XOR) function to createthe compound checksum aE[13:0] which will be transmitted in the nextframe. The compound 14-bit checksum aE[13:0] in the current frame isgenerated by an XOR operation of the “A” command 14-bit CRC from thisframe, with the latched FE[13:0] generated from the 72-bit data of theprevious frame.

To decode a frame at the receiving memory agent, a 14-bit commandchecksum CRC14[13:0] is generated from the 26-bit command, and a 22-bitdata checksum CRC22[21:0] is generated from the 72-bit data in thecurrent frame. CRC22[13:0] is latched as FE[13:0] for future compoundCRC checks in the next frame. A test compound checksum TESTaE[13:0] isgenerated through a bitwise XOR of FE[13:0] from the previous frame withthe new aE[13:0] from the current frame. If the generated test compoundchecksum TESTaE[13:0] matches the compound checksum aE[ 13:0]transmitted with the current frame, there are no errors in the “A”command of the current frame.

To complete the detection of faults in the 72-bits of data from theprevious frame, the 14-bit command checksum CRC14[13:0] generated fromthe current 26-bit command is XORed with the new aE[13:0] from thecurrent frame, thereby generating a result which is compared to thelatched FE[13:0] from the previous frame.

To start the fault detection of the 72-bits of data transferred in thecurrent frame, FE[21:14] transmitted with the current frame is comparedwith the new CRC22[21:14] generated from the 72-bit data in the currentframe. The completion of fault detection for the 72-bits of datatransferred in the current frame is done when the next frame arrives.

A fault in aE[13:0] indicates that both the “A” command in the currentframe could be faulted, and that the 72-bit data in the previous framecould be faulted. A comparison fault in the transmitted FE[21:14]partial checksum indicates that the 72-bit data in the previous framecould be faulted.

The CRC of the “A” command may be checked as soon as the first 4transfers of the frame are received and the information in the “A”command may be used immediately without waiting for the remainder of theframe to arrive. This mechanism may provide strong CRC protection of the72 data bits of the previous frame while reducing latency in thedelivery of the “A” command in the current frame.

The inventive principles are not limited to number or position of bitsshown in the embodiment described with respect to FIG. 31. The first andsecond portions of each frame have been referred to as command and dataportions, but any type of information may be transferred in eachportion, and the data portion may contain additional commands, commandextensions, etc. The tasks described do not necessarily need to beperformed in order described. Thus, the embodiment of FIG. 31 may bemodified to accommodate different system requirements or circumstances.

Some additional inventive principles of this patent relate to the use ofvariable mapping for testing lanes. FIG. 32 illustrates an embodiment ofa memory agent, in this example assumed to be a memory module or buffer,according to the inventive principles of this patent. The memory agent134 of FIG. 32 includes a receive link interface 140 which may have oneor more receivers and a transmit link interface 142 which may have oneor more transmitters. A loopback unit 196, which may be a multiplexer asshown here or other type of redirection device, is capable ofselectively mapping receive bit lanes to transmit bit lanes so that thememory agent may retransmit training sequences received from a memorycontroller back to the controller as return sequences on the transmitbit lanes. By selectively remapping receive bit lanes to transmit bitlanes using different mappings, the controller may analyze the returnsequences to identify not only whether there is a failed bit lane, butalso whether the failure is on a receive or transmit data path as wellas which receive or transmit bit position has failed.

The embodiment of FIG. 32 is shown with a receive and transmit linkinterfaces having unidirectional bit lanes, but the inventive principlesare not limited to this particular configuration of interfaces or typesof lanes.

FIG. 33 illustrates example embodiments of two possible bit lanemappings according to the inventive principles of this patent. Themappings shown in FIG. 33 assume, for purposes of illustration only,that the memory agent has 10 bit lanes in the receive link interface and14 bit lanes in the transmit link interface. Using mapping A, theloopback unit redirects the training sequences received on the lowerfive bits of the receive bit lanes to the transmit bit lanes such thateach of the lower five bit lanes is redirected to multiple transmit bitlanes. Using mapping B, the training sequences received on the upperfive bits of the receive bit lanes are retransmitted to the multipletransmit bit lanes.

A training sequence may contain a mapping indicator to instruct thememory agent which mapping to use. A training sequence may also containvarious groups of bit transmissions that provide test parameters to thememory agent or that provide electrical stress patterns that test thesignal integrity of each bit lane. Each of the bit lanes may receive thesame training sequences, or different bit lanes may receive differentsequences, for example, sequences having different electrical stresspatterns.

The training sequences received by the memory agent may be retransmittedwithout modification so that they function as the return sequences, orthe memory agent may modify the sequences or create entirely differentsequences. For example, the memory agent may retransmit most of thetraining sequence as the return sequence while modifying only a smallgroup within the sequence to provide identifying or status informationto the memory host.

If memory agents having multiple ports and variable mapping capabilitiesaccording to the inventive principles of this patent are utilized, forexample, in a multiple-agent configuration such as that shown in FIG. 3,the agents may be constructed so that only the outermost agent providesloopback operation, while the other agents operate in a pass-throughmode during a testing operation.

The embodiments described herein may be modified in arrangement anddetail without departing from the inventive principles. Accordingly,such changes and modifications are considered to fall within the scopeof the following claims.

1. A method for transmitting a frame from a memory agent comprising:transmitting a first portion of the frame; transmitting a second portionof the frame; and transmitting a CRC code for the first portion of theframe before the second portion of the frame is finished transmitting.2. A method according to claim 1 further comprising transmitting the CRCcode before transmitting the second portion of the frame.
 3. A methodaccording to claim 1 wherein the first portion of the frame comprisesthe CRC code.
 4. A method according to claim 3 wherein the first portionof the frame comprises the entire CRC code.
 5. A method according toclaim 1 wherein the first portion of the frame comprises a command.
 6. Amethod according to claim 1 wherein the second portion of the framecomprises a second CRC code for the second portion of the frame.
 7. Amethod according to claim 6 wherein the second portion of the framecomprises a portion of the second CRC code.
 8. A method for receiving aframe at a memory agent comprising: receiving a first portion of theframe; receiving a second portion of the frame; and receiving a CRC codefor the first portion of the frame before the second portion of theframe is completely received.
 9. A method according to claim 8 furthercomprising receiving the CRC code before receiving the second portion ofthe frame.
 10. A method according to claim 8 further comprising usingthe CRC code to check the first portion of the frame before the secondportion of the frame is completely received.
 11. A method according toclaim 10 further comprising using information in the first portion ofthe frame before the second portion of the frame is completely received.12. A method according to claim 8 wherein the first portion of the framecomprises a command.
 13. A method according to claim 12 wherein thesecond portion of the frame comprises a second command.
 14. A methodaccording to claim 12 wherein the second portion of the frame comprisesmemory data.
 15. A memory agent comprising circuitry to: transmit afirst portion of a frame; transmit a second portion of the frame; andtransmit a CRC code for the first portion of the frame before the secondportion of the frame is finished transmitting.
 16. A memory agentaccording to claim 15 wherein the circuitry may transmit the CRC codebefore transmitting the second portion of the frame.
 17. A memory agentaccording to claim 15 wherein the first portion of a frame comprises acommand.
 18. A memory agent comprising circuitry to: receive a firstportion of a frame; receive a second portion of the frame; and receive aCRC code for the first portion of the frame before the second portion ofthe frame is completely received.
 19. A memory agent according to claim18 wherein the circuitry may receive the CRC code before receiving thesecond portion of the frame.
 20. A memory agent according to claim 18wherein the first portion of a frame comprises a command.
 21. A memoryagent according to claim 20 wherein the circuitry may use the commandbefore the second portion of the frame is completely received.
 22. Amemory system comprising circuitry to: transmit a first portion of aframe; transmit a second portion of the frame; transmit a CRC code forthe first portion of the frame before the second portion of the frame isfinished transmitting; receive a first portion of a frame; receive asecond portion of the frame; and receive a CRC code for the firstportion of the frame before the second portion of the frame iscompletely received.
 23. A memory system according to claim 22 whereinthe circuitry may receive the CRC code before receiving the secondportion of the frame.
 24. A memory agent according to claim 22 whereinthe first portion of a frame comprises a command.